chore(streaming): re-sync #784 with main after #802, #811, #813 and #814 - #821

… through a pattern (#811)

* security: the ODCS and plugin-extension checks let a trailing newline through a pattern

The FLUID fix (schema_validator, #799) left six JSON Schema checks on
jsonschema's own pattern keyword, Python's re.search, whose $ also matches
before a final newline:

- forge/core/artifact_validators.py (fluid validate-artifacts: ODCS, Bitol ODPS)
- providers/odcs/validation.py::collect_errors (fluid odcs validate)
- providers/odps_standard/validation.py::validate
- providers/opds/validator.py (pinned Draft202012Validator)
- forge_datamodel/emit/osi_sidecar.py (the copilot's Ossie check)
- extension_schemas.py::assemble_proposed_extensions (pinned Draft7Validator)

So ODCS refused schema[].id "t 1" but took "t1\n", which an id: | block
produces, and the copilot kept an LLM-proposed customScaffold block with
use: "ci:basic\n", ref: "main\n" or id: "ci\n". All six now go through
schema_validator, and say so with the same line-break hint.

schema_validator takes a default dialect for a schema that declares none,
passed to jsonschema's validator_for(default=), the way OpenStack Rally
uses it. An extension schema is read in the dialect it declares, draft-07
when it declares none (the documented plugin contract), so a 2020-12
schema's unevaluatedProperties: false is enforced instead of ignored. OPDS
keeps 2020-12 as its default.

patternProperties keys keep re.search. jsonschema matches them in
patternProperties, additionalProperties and unevaluatedProperties, and
check-jsonschema's override of patternProperties alone lets a key that one
keyword matches and another does not through additionalProperties: false
unchecked. Only OPDS has key patterns (^[a-z]{2}$, three places). The
tripwires now cover every shipped schema, pin that set, and also fail on a
$recursiveRef or $dynamicRef, which re-pick the stock class as a nested
$schema does.

Follows #799.

* security: review follow-ups for the vendored-schema end-anchor change

- fluid odcs validate cuts each schema message to 120 characters, so for
  an id like customer_orders_v1 the hint ended "... block keeps and |",
  which reads as advice to use |. _brief cuts the message ahead of the
  hint instead; the hint is schema_manager.LINE_BREAK_HINT now, so the
  command can find it.
- schema_validator raises ImportError when jsonschema could not be
  imported. validate_osi_document, the one unguarded caller, raised
  NameError: jsonschema instead. validate_opds reads the same
  JSONSCHEMA_AVAILABLE flag as the other checks; the find_spec probe that
  found an unimportable jsonschema is gone.
- A test walks fluid_build/ and fails on any validator_for(),
  Draft*Validator() or jsonschema.validate() without cls= outside
  schema_manager, so a check added later cannot read $ as Python does
  again. It finds the six sites on the base and seven on the old main.
- Tests for the flag branches of collect_errors and the artifact
  validators, and for OPDS reading an undeclared schema as 2020-12.
- CHANGELOG: the extension check shows no message, so it carries no hint.
- odps_standard.validate validates against the schema for the document's
  apiVersion, not only v1.0.0.

* fix: a bracket in a value or an unusable plugin schema no longer stops the run

Defect-scan follow-ups for the vendored-schema end-anchor change:

- fluid odcs validate printed each schema error as Rich markup, so a
  failing value with "[/]" in it ended the command with "Unexpected
  error: closing tag '[/]' ... has nothing to close" (exit 2, also on
  main). The line is printed with markup=False, like the vowl line in
  the same function.
- assemble_proposed_extensions raised when a plugin schema's $schema was
  not a string: validator_for cannot look it up, where Draft7Validator
  ignored it. A schema that cannot be applied now drops its block with a
  redacted warning, as a block that fails its schema is dropped, and the
  run goes on. This also covers {"type": 5}, which raised on main.
- The guard test also catches `from jsonschema import validate`,
  `import jsonschema as js` with js.validate(...),
  jsonschema.validators.validate/extend/create, and a cls= that does not
  come from schema_validator. ODPS passes cls=type(schema_validator(...))
  inline so the guard can see it. It parses only files that mention
  jsonschema.

* docs(changelog): keep the two pattern end-anchor entries together

The rebase onto main put two new fluid diff --baseline entries between
the FLUID-schema entry (#799) and this branch's, which begins "The fix
above covered the FLUID schemas only".

* feat(validate,plan): warn about an id the build phase refuses

The schema's $defs.identifier admits ids the build phase refuses: a
leading digit (1orders) or more than 128 characters. fluid apply in a
build mode refuses such an id before it applies anything (#803), but
fluid validate and fluid plan said nothing, so the first sign was a
failed apply. The two grammars stay different by decision.

For a contract with builds, both now warn and name the id, using the
rule apply uses (build_runners._ids.validate_build_ids):

- validate adds a ValidationResult warning, so plain validate passes
  and --strict fails (founder's call). It runs with the other 0.7.x
  contract rules, on a contract file and on a bundle's resolved
  contract. If the check itself crashes, that is reported as a warning.
- plan prints the warning and still writes plan.json, whatever its
  --mode (founder's call). It runs after the gate plan shares with
  apply, not inside it, so apply keeps its typed CLIError.

IDENT_RULE states the grammar in words, and both the new warning and
apply's hint use it. The hint said validate and plan "accept" such an
id; it now says they only warn about it.

* fix(validate,plan): keep the build-id warning true on a schema-refused id

Review follow-ups to the build-id warning:

- validate runs its contract rules on a contract the schema refused too,
  so the warning could say "The contract schema allows more" about an id
  the schema had just refused (id: 123). The text now only says what the
  build phase needs and that fluid apply in a build mode refuses the id.
- plan prints the warning the way validate prints its warnings: literal,
  soft_wrap (the terminal wraps, so Rich never breaks a line inside the
  id) and no highlighter.

#802)

* fix(diff): compare column types strictly enough for --fail-on-breaking

Now that fluid diff --baseline reads 0.7.x columns, its type comparison
decides --fail-on-breaking for those contracts, and several type changes
that break consumers passed as widenings or as info.

- Fold case for ASCII type names only. Text the comparison cannot order
  (non-ASCII, or parameters that are not plain integers, such as
  enum('a','b') or varchar(max)) is its own type, so it no longer reads
  as unchanged, as "no parameters" or as a whitespace difference. The
  finding says so when a non-ASCII name is the reason.
- A parameter change that is neither a pure widening nor a pure
  narrowing is breaking (it was info): varchar -> varchar(10),
  decimal(10,2) -> decimal(12,1), decimal(10,2) -> decimal.
- A bigger parameter widens only for types whose parameter measures
  capacity (lengths, precisions, the decimal family), not for ones that
  name something, such as geometry(srid).
- decimal, numeric and number widen only when the scale and the integer
  digits (precision - scale) both hold.
- A change of base type widens only when the new type declares no
  parameters, so int -> decimal(5,0) is breaking.
- time -> timestamp, float -> decimal/numeric and int -> float leave the
  widening table.
- The type grammar is ASCII only and is matched in linear time.

Prior art: the Apache Iceberg table spec ("Schema Evolution": int ->
long, float -> double, date -> timestamp from v3, decimal(P,S) ->
decimal(P',S) for P' > P) and Delta Lake "Type widening" (decimal(p,s)
-> decimal(p+k1,s+k2) for k1 >= k2 >= 0).

* fix(diff): report a name the new contract declares twice

The comparison indexes exposes, columns and nested fields by name and
keeps the last of a repeated name, so an earlier declaration was never
compared against the baseline. An expose id, column name or nested field
name that the new contract declares more than once is now a breaking
change (expose_duplicate_id, column_duplicate_name,
nested_field_duplicate_name), in an expose that is new as well as in one
both contracts have, and whatever the baseline holds: compare_contracts
is idempotent except for a contract that declares a name twice. The
declarations are in the finding's "after".

* test(diff): diff the schema-invalid fixtures with validate=False

fluid diff --baseline validates both contracts since #805, so a fixture
that is not a valid contract (the flat-layout and malformed-schema ones)
is diffed with validate=False, as the neighbouring tests are.

A plain merge of origin/main (24f1773). Since #815, main is a real
ancestor, so git's base (635ea30) is the right one.

Resolutions, two conflicts:
- fluid_build/build_runners/_deployment_host_policy.py:
  - #814 already carries #809's `VerdictCode`, `HostVerdict` (with
    `pinned`), `host_verdict`, `pinned_verdict` and `host_refusal`; their
    ASTs are identical on both sides.
  - The conflict was #809's `is_listed`, which main lacks. Kept.
  - Main's #814 guard (`_host_error`, `_pinned_refusal`,
    `refuse_unless_host_allowed`, `request_guard`) is unchanged.
- pyproject.toml: #814 adds the same two mypy-strict modules and comment as
  #809. Kept #809's third module, `fluid_build.streaming.policy`. The file
  now equals the feature branch's.

Both sides also changed these files, and git merged them without conflict:
- changelog_rules.py: only `diff_columns` changed on both sides: #802's
  type-change wording, and #784's `_diff_container_children` call.
- plan.py: only `run()`: #813's `_warn_unusable_build_ids` runs after the
  plan gate, ahead of #784's streaming routing.
- apply.py and schema_manager.py: no function changed on both sides.

In those four files each side's diff is unchanged. The tree differs from
main in exactly the feature's 44 files.

#811 added a guard, test_every_validator_is_built_by_schema_validator,
which fails on any validator built outside schema_manager.schema_validator.
streaming/compiler.py built a plain Draft202012Validator for
parameters.flink, so the merged tree failed that guard.

Building it through schema_validator keeps Draft 2020-12 as the dialect
and makes each pattern end at the end of the value. So a trailing newline
in deployment.name or image is now refused; #809's allowlists did not cover
those two fields. Two test_fail_closed cases pin it (red before this change).

The feature branch moved during review: #817 (Kafka record shapes declared
in FLUID) was squash-merged onto 0b4fc36 as a89966a. Merging it keeps this
sync mergeable into the feature branch. The base is 0b4fc36.

Resolutions, three hunks in two files:
- fluid_build/api/changelog.py, compare_contracts: kept both. #817's
  `diff_streams` stays after the sovereignty rules, where #817 put it.
  Main's `diff_duplicates` (#802) sits directly above the per-expose
  comparison its comment refers to.
- fluid_build/streaming/compiler.py imports: kept main's `schema_validator`
  and #817's `streaming.mapping` names.
- fluid_build/streaming/compiler.py, compile_build: kept #817's refusal of
  topic, format and schema in parameters.flink.source, which now live in
  properties.stream. Then the config is validated through `schema_validator`
  (3fa5f4c) in place of #817's raw Draft202012Validator.

tests/streaming/test_compiler.py merged without conflict.

In those three files each side's diff is unchanged. The tree differs from
a89966a in exactly main's changes since 635ea30 plus 3fa5f4c's two files.

Brings in #820 (litellm 1.108.0's provider endpoints are dotenv-protected), which fixes the endpoint-names drift test that turned every PR's Test legs red. A clean merge: #820 touches CHANGELOG.md and credentials/litellm_endpoints.py, neither of which the feature changes. The merge adds exactly #820's patch to 00e73e2.